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We investigate the impact of integrating dynamic geometry environments with haptic devices, 
which allow users to not only see and manipulate geometric figures on a screen but also feel, 
through force-feedback, the result of interacting with such objects. The feedback loop can be 
assigned to varying attributes of objects, for example, the changing area of a deformable 
triangle, as well as invariant ones. We report preliminary findings from our work in informal 
settings. Our findings indicate that such experiences allow young children as well as 
undergraduates to yield expressive discourse that is intimately connected to the mathematical 
concepts being presented. 


Introduction 

Our project combines various technological ingredients to develop a new mathematical and 
scientific learning environment for allowing more students to access conceptually demanding 
ideas through various senses in K-5" grade classrooms. We are using a modified version of The 
Geometer’s Sketchpad® software, the H3D Application Programming Interface and specific 
haptic devices. 

Haptic literally means “ability to touch” or “ability to lay hold of’ (Revesz, 1950) and has 
evolved in a technological era to be an interface for users to virtually touch, push, or manipulate 
objects created and/or displayed in a digital visual environment (McLaughlin, Hespanha, & 
Sukhatme, 2002). Haptic devices can provide force-feedback or tactile feedback. Haptic 
interfaces allow users simultaneous information regarding their input and reaction to that input as 
programmed within a computational environment. 

The activities we are creating allow users to construct, interact and explore geometric figures 
and shapes, and so engage in model-eliciting activities in various mathematical topics. Our study 
is assessing and evaluating what new or enhanced learning experiences can be created by the 
synergistic integration of dynamic geometry with new haptic hardware, why this is necessary and 
how it can improve present practice. 

We are designing new mathematical investigations that build on established curricular 
standards and frameworks, and assess the potential scalability of such an approach in the future 
given these intersecting research traditions. We can finally do this because of the availability of 
new affordable devices, but such technological advances are irrelevant without an aim to 
transform the mathematical and scientific activities of the classroom. The first stage of our 
project, which we report on here, investigates the educational potential for introducing such a 
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multi-modal environment in different settings by investigating how it can engage different aged 
students to investigate mathematical ideas focused on change and variation. We particularly 
focused on features of expressive discourse that are elicited when students interact with our 
dynamic haptic geometry environment. We have analyzed whether the added modality of force- 
feedback can be engaging and rich for students in terms of how they can potentially discover 
new concepts, which has led to the emergence of design principles for future implementation in 
formal mathematics settings. 


Theory 

Haptic technology has evolved over the past 10 years, particularly out of a focus on virtual 
reality in the 90s, and has become more available in a variety of commercial and educational 
applications, including 3D design and modeling, medical, dental and industrial applications. 
However, research specifically examining haptics in relation to learning and education is scarce. 
A large proportion of the existing studies focus on “haptic perception” a major field in 
psychology focused on haptic sense, and the second main set was focused on multimodality. 
Multimodality reaches into education in various ways intersecting deeply with a multi-media 
approach. Recently, multimodal approaches have also focused on the role of gesture with 
increasing interest (Wagner Cook, Mitchell, Goldin-Meadow, 2008) with particular focus on this 
mode as a form of mathematical expressivity (Hegedus & Moreno-Armella, 2008). 

Historically, this has been translated as a way to create multiple learning pathways for 
students to work within auditory and visual modalities. In fact, the audio/visual modality is still 
the predominant “multi-” media form. But, students can interpret visual, auditory and haptic 
displays to gather information, while using their proprioceptive system (making sense of the 
relative positions of one's own body parts) to navigate and control objects in their synthetic 
environment (Dede, Salzman, Loftin & Sprague, 1999). In this work, multiple sensory 
representations can offer mutually reinforcing information that a user can collect to develop an 
understanding of a mathematical or scientific model. In addition, haptics have been said to be 
superior to vision in the perception of properties of texture and microspatial properties of pattern 
(Zangaladze, Epstein, Grafton, & Sathian, 1999), while vision is more useful in the perception of 
macrogeometry particularly shape and color (Sathian, Zangaladze, Hoffman, & Grafton, 1997). 
Leveraging these relative strengths, our proposed activities will incorporate the need to 
investigate both shape and properties of construction (i.e., the mathematical structure of a figure 
or surface). We propose that it is relevant to integrate haptic technology with dynamic geometry 
software to offer multiple sources of information-feedback for students; it is not enough to offer 
a way for students to just see a mathematical object or a scientific model in a static way they 
must also engage with it dynamically, tactically and naturally. 


Setup of Multi-Modal Technology Environments 

Description of the haptic activity setup 

The SenseAble PHANTOM Omni haptic device (See Figure 1) is capable of reading position 
and orientation data of a stylus in 3D space, which serves as the physical interface to the device. 
The device also provides force feedback to the user through the stylus. Users can grip the stylus 
and move it freely in 3D space with their choice of grip suitable to their intentions at any given 
time. We paired PHANTOM Omni with a laptop so that users are able to view a visual 
simulation as they simultaneously manipulate the device. Interviewers used the mouse and a 
separate keyboard to manage switching among activities and manipulating the parameters of 
activity variations. 
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Figure 1. The SenseAble Phantom Omni haptic device 


Description of the Types of Simulations 

We aimed to elicit student expressivity associated with two types of simulations: breaking 
point and impact. A breaking point is defined as a point of resistance against the haptic stylus, 
where the resistance is removed once a large enough force is exerted on the stylus. An impact is 
defined as bumping into, or being bumped by objects within the haptic environment. These 
simulations were embedded into activities in which the students were not initially presented with 
any visuals. This protocol gave us the opportunity to hear students draw heavily on their haptic 
experiences without the additional influence of the visual simulation. Later, visuals were 
revealed in order to examine ongoing expressive discourse in the presence of additional stimulus. 

One activity we embedded the simulation of breaking point is called Falling Off: Haptic 
experience in this activity consists of a resistant force occurring from a user's contact with a 
visual surface using the haptic device. As the user moves off the edge of the surface, the 
resistance is removed, which results in a feeling of falling off. Another activity involving the 
simulation of breaking point is called Break Through, wherein users encounter an unavoidable 
barrier in the center of the haptic environment. When contacted by the user, the barrier provides 
a resistant force, which can be removed once the user exerts sufficient force on the barrier. 
Additional haptic experiences can be added to either side of the barrier through the exertion of 
forces that oppose a user's movements within the haptic environment. 

In, Jmpact, our second simulation type, students encountered two types of obstacles as they 
explored the haptic environment. The Spheres activity presented a stationary 8x8x8 lattice of 
spheres. Spaces among the spheres allow users to probe around and through the lattice, bumping 
into the spheres in the process. The second activity involving the simulation of impact is called 
Bumped. In this activity, the obstacle moves perpetually and repeatedly from left to right with a 
speed controlled by the researcher. The users feel being bumped by the obstacle when they 
explore the center of the haptic environment. 


Methods 

During the spring of 2010, we collected preliminary observational data in the form of video 
and field notes from 32 students (27 elementary and five undergraduate) interacting with the 
haptic activities. The students involved were chosen from three different populations within the 
Southcoast region of Massachusetts including local elementary schools, university freshmen and 
young children from a local urban city Boys and Girls Club of America. The observations 
occurred during informal interview sessions, where researchers asked questions to the students 
about what they thought and fe/t and how they related these experiences to the dynamic visuals 
on the screen. For undergraduate students, the interview sessions lasted about an hour; for 
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elementary students, they lasted between 25-40 minutes. In this paper, we do not make a 
distinction among these students with respect to their developmental differences. Instead, we 
only provide a descriptive approach towards their haptic experiences to explore the features of 
their expressive discourse. Pseudonyms for student names are used. 

We used the qualitative software product NVivo” to examine the use of expressive discourse. 
For example, we addressed the following: How did the student dialogue cohere with the 
expectations of the activity? What can we learn about the types of interview question and 
response patterns that inform future interface and activity design? We have used NVivo’ to code 
for these types of expression as attributes for our particular cases (both students and activities). 
In order to answer the questions surrounding the activities’ expectations the coded student 
discourse was cross-referenced with the set of expectations for each particular activity. 

Our analysis of students' expressive discourse focused on how they could differentiate haptic 
experiences (e.g., “this felt like the ball was moving faster”). We initially identified three 
categories of expressive discourse. The first category is metaphorical word use. We define 
metaphorical word use as the set of utterances about objects or actions that are not present in the 
haptic-visual environment, but result from students’ previous experiences. We further classified 
the use of metaphorical word use as scholastic (mathematical/scientific) or non-scholastic (non- 
mathematical/scientific). Scholastic use of metaphors relates objects or actions to mathematical 
or scientific objects students have previously interacted with. Non-scholastic use of metaphors is 
not related to mathematical/scientific objects but to students' everyday experiences. 

The second category of expressive discourse is reaction to sensual experiences. We define a 
sensual experience as any form of expressivity that is a result of the students’ sensations and 
reactions to the feedback (e.g., force, sound, or visual) they receive from the haptic device. We 
subdivided this category into descriptive aesthetics, evaluative aesthetics, and gesturing. 
Descriptive aesthetics pertains to utterances about the color, size, or feel of what students are 
seeing, feeling or hearing (Sinclair, 2004). Evaluative aesthetics differs from the descriptive 
aesthetics in that, instead of just describing the form (e.g. it's green), students are assigning some 
affective statement to their experiences (e.g. it's scary or it's beautiful). In addition to the verbal 
reactions pertaining to an activity’s aesthetics, we also focused on students’ sensual reactions 
through gesturing. We consider gesturing as a set of bodily movement for communicating in 
context, or collaboration with speech; either to oneself, a fellow student, or one of the 
interviewers. 

The final category of expressive discourse is developing a way of measuring and calibrating 
visual and physical spaces. This corresponds to the students' routines as they attempt to measure 
and calibrate the physical and visual space during their haptic experience. 


Results 

Metaphorical Word Use 

Examples of scholastic metaphorical word use were seen during an activity where students 
were presented a flat surface, which they could drop onto a small, white ball. When asked what 
would happen when the surface was dropped, one elementary school student (aged 8), Annie, 
predicted that it would fall to one side. This prompted another similarly aged student, Beatrice, to 
relate it to a “see-saw”, and the white ball as the fulcrum. When asked to give a definition of 
fulcrum, Beatrice said “the center that keeps it together, like scissors”. More metaphorical word 
use was seen with elementary school students during the Break Through activity. Students were 
presented with a barrier that provided resistance to passing through it, until enough force was 
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exerted using the haptic stylus, thus releasing the resistance. As an explanation for this barrier, 
which some students described as a wall or “pressure”, two groups of students reasoned that it 
had magnetic properties that “[didn’t let] you go where you want”. Undergraduate participants 
also used expressions containing mathematical metaphors. In one activity three undergraduates 
were shown a shallow, concave parabolic curve. The haptic stylus controlled a point displayed 
on the x-axis and, as they moved the point in the positive direction, students felt an increasing 
force, proportional to the parabola’s slope, in the negative x-direction. In one student’s 
explanation of this force-feedback, he referred to the feeling as “riding across sine waves”. 

The use of non-scholastic metaphorical word use, however, was more prevalent in our 
interviews. In one Break Through activity, when the force opposing user movement was 
activated, students described their movements as “heavier”, “like rubber” or an “eraser on 
paper”. In contrast, movements without the force were described as “lighter” or “freer”. One 
student even compared their movements with, and without the force to being chased in a scary 
movie. The student related the opposing force to when they are walking and “you feel all that 
pressure, like someone’s behind you, and you’re scared”. Movement without the force present 
represented a “relief” for the student. 

Another metaphor discussed in a Break Through activity, related the force opposing 
movement to walking on a windy day. In the metaphor, the force was described by an elementary 
school student as “wind actually blowing you” towards the barrier. In a separate session, an 
undergraduate used the same metaphor, saying the force was like wind when you’re walking, 
“there would be resistance to your movements.” The Bumped activity also provided similarities 
between the elementary and undergraduate students. Two groups of students, one elementary and 
one undergraduate, described the periodic impacts as the beating or pumping of a heart, or a 
pulse. Along with the theme of periodicity, these two age groups also likened the period impact 
to a clock. 


Reaction to Sensual Experiences 

This type of expressive discourse classifies the communications of student reactions to 
sensual experiences during our activities. As an example, we again reference the force opposing 
movement in the Break Through activity. One student explained that when the force is not active 
the haptic environment “feels like it’s empty”, but with the force the environment has 
“something inside it.” This explanation makes a description, in terms of a container, about the 
haptic sensation associated with moving through the environment. 

The Spheres activity intends for students to describe how a lattice of spheres hindered or 
modified their movement within the haptic environment. During the interviews with both 
elementary and undergraduate students, movements across, and within the lattice were described 
as bumpy 20 times. Again, these “bumpy” descriptions pertain to student reactions to sensations 
caused by their movement through the environment. 

Evaluative aesthetics differs from the descriptive brand, because students make affective 
statements rather than just talking about the forms of their sensual experiences. For example, in 
the Break Through activity, when one student pushed against the barrier he described it as 
“weird”. When further evaluating the experience he said, “it feels like rubbing on rubber. ..it’s 
weird.” Some elementary school students exploring the spheres in the Spheres activity began to 
associate the colors of the spheres with how “hard” they felt. Interestingly, however, the stiffness 
of the spheres in the activity was identical. In the Bumped activity, one group of students 
indicated that the impact from the moving obstacle was emotionally affective. The students said 
that the unexpected impacts became “scary”, because the sphere “just jumps out of nowhere’. 
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In addition to the verbal reactions pertaining to an activity’s aesthetics, students in our 
interviews also expressed their sensual reactions through gesturing. The Bumped activity 
provided a number of gestural reactions, mainly because it was the only activity offering objects 
that were out of the students’ control. In each of our other activities, students were the initiators 
and producers of sensation on objects in the haptic environment. However, the students could not 
control the moving obstacle in Bumped. Therefore, students would make gestures to either avoid 
coming into contact with the obstacle, or seek out contact with it. This type of gesturing occurred 
when the obstacle was both invisible and visible. 

In one interview, a student named Catie discovers that there is a different haptic sensation 
that results from pulling the device close to her body, rather than towards the base of the device. 
When the next student, David, begins using the device, Catie gets the attention of David and 
makes a gesture, pulling her arm towards her body, in an effort to show David the difference. 
While Catie makes the gesture, David mimics it, presumably in an attempt to achieve the same 
sensation as Catie. 

One student, while moving through the Bumped activity used a gesture to test if they were 
causing the impacts or if the impacts were independent of their movement. While the student, 
Ethan, was moving the haptic device he noticed the device providing force-feedback. Ethan 
claims that he is not the source of the feedback, saying the device is “doing it on its own.” 
Another student, Fred, questions Ethan by saying, “how do we know you’re not touching it.” In 
response, Ethan rests the haptic stylus in the palm of his hand, looks at Fred, and says, “see, I’m 
not doing it!” Fred appears to be satisfied with the evidence. This episode is quite significant 
because it illustrates a gesture powerful enough to convince another student that the sensation 
felt by Ethan was not caused by Ethan’s movements. 


Developing a way of measuring and calibrating visual and physical spaces 

In some cases, fifth-grade students compared the regular structure of the lattice of spheres 
within the Spheres activity to familiar objects with regular structure. Students most often 
described the spheres as discrete objects, such as “bumps” or “rocks.” As they explored, students 
also characterized the arrangement of the objects in space, starting with horizontal structure and 
then adding in a vertical component. One group offered, “it feels like you’re going over bump 
after bump,” noting repetition left to right. They later decided “it feels like stairs because it’s 
going higher and higher,” adding in the vertical. In this group, “stairs” became an accepted 
description adopted among the students and repeated: “like going down stairs,” “like rocks piled 
up as stairs.” Another group, that followed a similar pattern of describing horizontal and vertical 
structure, was more systematic in their investigations and specific in describing how the 
horizontal and vertical structure interrelated. Moving from left to right, a student sought to 
describe the repetition additively: “If I move one over, there's one more, and one more over 
there. Once you go over here, there's no more.” This group also explored the vertical pattern in 
conjunction with the horizontal, using stairs as a way to describe change in two dimensions: 
“Every time I move down there's something stopping, and up, and sideways, kinda. [...] As you 
go up, it stops, up, stop. [...] It’s kinda like stairs, you stop it goes down, stop it goes down.” This 
group also adopted “stairs” as one shared way to describe one aspect of the structure of the 
matrix. 

The use of stairs to relate vertical and horizontal structure appeared during the non-visual 
portion of the Spheres activity. Once the students were shown the sphere lattice, one face of this 
three dimensional lattice was immediately revealed to them. The nature of flat computer 
monitors meant that the depth of the 3D model was obscured by the front of the lattice. But, 
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students still noticed the depth in their explorations with the haptic stylus and in the visuals. In 
one third-grade group, Doug, Ellen and Fiona wondered aloud about the structure of this 
simulation in the depth dimension. By counting the visible spheres and using multiplication, 
Fiona decided that a face had 35 spheres (the proximity of the virtual camera to the lattice meant 
that some of the spheres were out of the field of view, and students counted a 5 x 7 lattice in the 
front face). As Ellen began to count the spheres by tapping them with the haptic stylus, Doug and 
Ellen agreed that the front face was not the entirety of the lattice. “35 is just the top.” They 
sought a way to complete the count in the depth dimension, which was more difficult to see on 
the screen. Ellen said, “You can't tell how many there are down, so you can't tell how many there 
are.” She went on to elaborate as she explored in the depth dimension: “If you, like, move it 
farther away and closer to the machine you can kind of feel the ones on the bottom, but not 
really.” The laptop screen was tilted away from her, and her descriptions appear to associate 
“down” with depth. As she continued to probe, they discussed how one might arrive at a full 
count of the spheres. “You would do 37, I mean 35 times how many there is down.” While Doug 
attempted to tell how many there were based on the shadows of the spheres in the display, Ellen 
began counting aloud. As she moved the stylus closer, and then farther away from herself she 
counted: “One, two, three, four...” When asked whether she was counting the spheres on the 
laptop screen, she said, “No. Feeling them.” It appeared that the desire to get a full count of the 
lattice and the difficulty of getting accurate visual confirmation led Ellen to count by feeling 
variation along one specific dimension. Communication among the students and the researchers 
during this activity involved both expressive word use and gesture, as Ellen used deliberate and 
exaggerated poking motions to illustrate her count while we all attended to her efforts. 


Discussion 

From the findings that relate the expectations of each activity and students’ expressive 
discourse, we formulated a set of haptic design principles. For example, activities that provide 
resistance in the form of a breakable wall (i.e. resistance against moving then, after a certain 
level of force is exerted, no resistance) are described by students as containing stoppages, 
“getting stuck” and preventing them from “going where you want”. We also discovered that 
students were able to discriminate varying levels of viscosity, when the viscosity change is 
preceded by a breaking point. 

No visuals in an activity often led to students exploring positions within the haptic 
environment where they posited haptic sensations would occur. Student discourse also indicated 
that haptic sensations could incite affective statements. For example, when students encountered 
an object moving along a track, the frequency of impact led to unsettled emotions (e.g. “creepy”, 
“scared”, “losing control”). 

The analysis also led to the identification of distracters in the activities, defined as any 
element of an activity (i.e. visual or haptic) that distracts the user from the design intentions of 
the activity. For example, in the Falling Off activity, students were more focused on the visible, 
squiggly lines on the surface instead of exploring the feeling of moving off the surface. Further, 
we found that students who worked within haptic activities more than once were engaged (i.e. 
attentive to the interview questions and activity tasks) throughout the 25-30 minute interview. In 
fact, some students were more active as their interaction with the haptic environment increased. 

In creating haptic principles and identifying distracters, we have now discovered a set of 
haptic experiences that have been shown to be successful in eliciting certain student discourse. 
Using this knowledge, we are designing more mathematical haptic activities, in preparation for a 
main study during the spring of 2011. 


Wiest, L. R., & Lamberg, T. (Eds.). (2011). Proceedings of the 33rd Annual Meeting of the North 
American Chapter of the International Group for the Psychology of Mathematics Education. 
Reno, NV: University of Nevada, Reno. 


Articles published in the Proceedings are copyrighted by the authors. 


PME-NA 2011 Proceedings 619 


Students' expressive discourse was particularly rich in their reaction to sensual experiences 
and use of metaphors. Students’ genuine participation in the experience was reflected in their 
gestures and utterances of surprise, excitement, shock, etc. which provided a natural context in 
which they developed and tested self-generated hypotheses. The instances in which students 
shifted their everyday metaphors to scholastic (mathematical/scientific) ones made us wonder the 
characteristics of the contexts in which they moved between the informal and formal discourse 
during their experiences. Since our future goal is to use the haptics environment to develop and 
implement mathematical activities, adding a fourth category to our initial classification of 
expressive discourse seems necessary. The fourth category would mainly focus on how, and 
under what conditions, students move back and forth between everyday words to the more 
technical/formal, mathematical words. We are also interested in how students cope with the 
notions of variance and invariance as well as continuity and discreteness as they conjecture about 
mathematics through their haptic experiences in physical environments. 

In summary, a new set of design principles emerged from our analysis of expressive 
discourse as we take steps towards making the haptics experience mathematically relevant and 
significant. We believe that students! self-motivated hypothesizing/theorizing through interaction 
with peers, as reflected by their discourse in our preliminary study, is crucial for mathematics 
learning. The haptics environment presents many opportunities to contribute to the teaching and 
learning of mathematics. 


References 

Battista, M.T. (1994). Teacher beliefs and the reform movement in mathematics education. Phi 
Delta Kappa, 2, 462-469. 

Dede, C. Salzman, M. C., Loftin, R. B., & Sprague, D. (1999). Multisensory immersion as a 
modeling environment for learning complex scientific concepts. In N. Roberts, W. Feurzeig 
& B. Hunter (Eds.), Computer modeling and simulation in science education (pp. 282-319). 
New York: Springer-Verlag. 

Freudenthal (1983). Didactical phenomenology of mathematical structures. Holland: D. Reidel 
Publishing Company. 

Hegedus, S., & Moreno-Armella, L. (2008). Analyzing the impact of dynamic representations 
and classroom connectivity on participation, speech and learning. In L. Radford, G. 
Schubring & F. Seeger (Eds.), Semiotics in mathematics education: Epistemology, history, 
classroom and culture (pp. 175-194). Rotterdam, The Netherlands: Sense Publishers. 

McLaughlin, M., Hespanha, J., & Sukhatme, G. (2002). Touch in virtual environments: Haptics 
and the design of interactive systems. Upper Saddle River, NJ: Prentice Hall. 

Minogue, J., & Jones, M. G. (2006). Haptics in education: Exploring an untapped sensory 
modality. Review of Educational Research, 76(3), 317-348. 

Revesz, G. (1950). The psychology and art of the blind. London: Longmans Green. 

Sathian, K., Zangaladze, A., Hoffman, J., & Grafton, S. (1997). Feeling with the mind’s eye. 
Neuroreport, 8, 3877-3881. 

Sinclair, N. (2004). The roles of the aesthetic in mathematical inquiry. Mathematical Thinking 
and Learning, 6(3), 261-284. 

Wagner Cook, S., Mitchell, A., & Goldin-Meadow, S. (2008). Gesturing makes learning last. 
Cognition, 106(2), 1047-1058. 

Zangaladze, A., Epstein, C. M., Grafton, S. T., & Sathian, K. (1999). Involvement of visual 
cortex in tactile discrimination of orientation. Nature, 401, 587-590. 


Wiest, L. R., & Lamberg, T. (Eds.). (2011). Proceedings of the 33rd Annual Meeting of the North 
American Chapter of the International Group for the Psychology of Mathematics Education. 
Reno, NV: University of Nevada, Reno. 


Articles published in the Proceedings are copyrighted by the authors. 


